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Abstract 

The  first  part  of  this  manuscript  examines  the  impact  of  configuration  changes  to  the  learning 
curve  when  implemented  during  production.  This  research  is  a  study  on  the  impact  to  the 
learning  curve  slope  when  production  is  continuous  but  a  configuration  change  occurs. 
Analysis  discovered  the  learning  curve  slope  after  a  configuration  change  is  different  from  the 
stable  learning  curve  slope  pre-configuration  change.  The  newly  configured  units  were 
statistically  different  from  previous  units.  This  supports  that  the  new  configuration  should  be 
estimated  with  a  new  learning  curve  equation.  The  research  also  discovered  the  post¬ 
configuration  slope  is  always  steeper  than  the  stable  learning  slope.  Secondly,  this  research 
investigates  flattening  effects  at  tail  of  production.  Analysis  compares  the  conventional  and 
contemporary  learning  curve  models  in  order  to  determine  if  there  is  a  more  accurate  learning 
model.  Results  in  this  are  inconclusive.  Examining  models  that  incorporate  automation  was 
important,  as  technology  and  machinery  play  a  larger  role  in  production.  Conventional  models 
appear  to  be  most  accurate,  although  a  trend  for  all  models  appeared.  The  trend  supports 
that  the  conventional  curve  was  accurate  early  in  production  and  the  contemporary  models 
were  more  accurate  later  in  production. 

Introduction 

The  Budget  Control  Act  of  201 1  subjected  the  Department  of  Defense  (DoD)  to  a 
more  fiscally  constrained  and  financially  conscious  environment  than  ever  before, 
juxtaposed  with  a  demand  for  new  aircraft  programs  of  almost  every  type.  As  an  increasing 
number  of  programs  are  terminated,  with  budget  overruns  being  a  contributing  factor, 
managers  at  every  level  in  the  DoD  are  expected  to  ensure  the  Department’s  shrinking 
budget  is  being  used  in  the  most  effective  way.  The  increased  scrutiny  adds  greater 
emphasis  on  the  accuracy  of  program  office  cost  estimates  given  that  an  approved  program 
cost  estimate  supports  every  major  aircraft  acquisition  program  funded  by  the  Department. 

The  current  state  of  the  DoD  includes  shrinking  budgets  and  large  funding  cuts  for 
acquisition  programs.  An  extra  emphasis  on  scrutiny  of  accurate  cost  estimates  is  the  result 
of  the  current  cuts  and  budget  issues.  There  is  a  new  standard  for  Financial  Managers  and 
Program  Managers  who  have  to  support  and  maintain  a  cost  estimate  like  no  time  before. 
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Having  a  balanced  budget  is  a  concern  for  the  DoD,  and  the  budget  depends  on  the  cost 
estimate.  Gone  are  the  days  when  the  DoD  had  ever-growing  budgets  where  the  fiscal 
mentality  was  to  spend.  The  fiscal  mentality  now  involves  saving  and  receiving  as  much 
value  as  possible  for  every  budgeted  dollar. 

In  order  to  obtain  reliable  cost  estimates,  cost  estimating  models  and  tools  within  the 
DoD  present  the  opportunity  for  an  evaluation  on  their  accuracy.  The  current  learning  curve 
methods  within  the  DoD’s  cost  estimating  procedures  are  from  the  1930s.  As  automation 
and  robotics  increasingly  replace  human  touch-labor  in  the  production  process,  a  model  that 
is  80  years  old  and  assumes  constant  learning  may  no  longer  be  appropriate  for  accurate 
learning  curve  estimates.  Robotics  and  automation  do  not  learn,  and  they  are  inevitably  a 
part  of  future  production.  New  learning  curve  methods  that  consider  automated  production 
should  be  examined  as  a  possible  tool  for  cost  estimators  to  utilize.  The  modern  learning 
curve  methods  could  be  a  useful  tool  for  obtaining  better  cost  estimates  within  the  DoD.  The 
purpose  of  this  research  ultimately  is  to  investigate  new  learning  curve  methods,  develop 
the  learning  curve  theory  within  the  DoD,  and  pursue  a  more  accurate  cost  estimation 
model. 


A  vital  input  to  the  cost  estimate  for  a  production  program  is  the  assumed  learning 
curve  slope  for  the  program.  The  learning  curve  often  depicts  the  learning  phenomenon  that 
occurs  in  manufacturing.  Learning  is  defined  as  a  constant  percentage  reduction  of  the 
required  touch  labor  hours  (or  costs)  to  produce  an  individual  unit  as  the  quantity  of  units 
produced  doubles  (Yelle,  1979,  p.  302) — as  the  number  of  units  produced  doubles,  the 
number  of  hours  required  to  produce  a  single  unit  decreases  by  the  learning  curve  rate. 
Learning  is  also  defined  as  both  the  conceptual  and  the  physical  learning  of  a  physical 
process  (Watkins,  2001,  p.  18).  The  learning  curve  for  a  program  is  generally  considered 
stable  once  the  program  is  substantially  into  production  because  the  manufacturer  and 
laborers  have  produced  enough  units  to  learn  the  most  efficient  production  process. 
However,  intuitively  and  through  past  research,  it  is  known  that  learning  is  disrupted  by 
changes  in  production,  and  only  the  production  of  additional  units  can  recover  the  lost 
learning  (Watkins,  2001,  p.  18).  It  is  critical  to  capture  the  change  in  the  learning  rate  due  to 
production  modifications  to  better  estimate  DoD  program  costs. 

This  comparative  analysis  study  will  examine  whether  different  learning  curve 
models  are  more  accurate  than  Wright’s  Learning  Curve  model  (the  status  quo)  when 
comparing  actual  values  to  predicted.  The  current  DoD  learning  curve  methodology  does 
not  take  into  account  available  information  and  factors  that  contribute  to  learning.  The  point 
of  emphasis  for  this  research  and  the  issue  that  needs  to  be  resolved  is  that  DoD  agencies 
need  to  estimate  more  accurately.  Prior  research  on  this  subject  shows  that  the  learning 
curve  methods  have  room  for  further  development.  There  may  be  an  opportunity  to 
incorporate  alternate  learning  curve  models  and  more  DoD  programs  into  this  area  of 
research.  Research  found  that  an  important  factor  (incompressibility)  was  not  explicitly 
researched  or  known.  Towill  and  Cherrington  defined  incompressibility  as  the  percentage  of 
the  learning  process  that  is  automated  (Towill,  1990).  Robotics  and  automation  are  not 
going  away  and  will  likely  play  a  larger  role  in  the  future.  Research  on  what  that  factor 
actually  equals  or  how  it  relates  to  different  airframes  could  be  critical  for  obtaining  a  more 
precise  model.  Using  integration,  assembly,  and  checkout  (IA&CO)  processes  instead  of 
complete  touch  labor  processes  should  provide  an  analysis  that  is  more  insightful  and 
potentially  leads  to  a  more  accurate  model.  IA&CO  are  specific  work  that  occurs  during 
production. 

The  idea  of  learning  in  a  production  environment  is  well  established.  T.P.  Wright  first 
published  the  learning  curve  phenomenon  in  early  1936.  Wright  observed  that  in  a 
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manufacturing  environment,  as  the  cumulative  quantity  of  units  produced  doubled,  the 
cumulative  average  cost  decreased  at  a  constant  rate  (Wright,  1936,  pp.  124-125;  Yelle, 
1979,  p.  302).  During  World  War  II,  government  contractors  investigated  the  usefulness  of 
the  learning  curve  concept  to  predict  labor  hours  and  cost  requirements  for  aircraft  and  ship 
construction  projects.  The  private  sector  went  on  to  adopt  the  learning  curve  theory  into 
practice  shortly  thereafter.  Although  learning  curve  theory  has  evolved  and  has  been 
referred  to  by  different  names  in  the  decades  following  Wright’s  report,  including  the 
experience  curve,  the  progress  curve,  and  the  improvement  curve,  Wright’s  model  remains 
one  of  the  models  most  widely  used  by  manufacturers  to  predict  labor  hours  and  costs 
(Yelle,  1979,  pp.  303-304;  Badiru,  1992,  p.  176.). 

Wright’s  original  findings  postulated  a  constant  learning  environment;  however, 
researchers  have  not  ignored  the  idea  that  constant  learning  may  not  exist  on  a  continual 
basis  in  a  manufacturing  environment.  In  fact,  the  ideas  of  regressed  and  lost  learning  have 
been  widely  studied.  Research  studies  support  that  a  break  in  production  creates  an 
environment  of  relearning  because  the  labor  resources  have  stopped  working,  at  least  on 
the  same  project,  and  will  be  less  efficient  at  manufacturing  when  production  restarts 
(Anderlohr,  1969,  pp.  16-17). 

In  addition  to  production  breaks,  instances  also  exist  when  a  major  configuration 
change  occurs  during  production  and  disrupts  the  learning  process.  In  this  situation,  the  new 
configuration  is  immediately  incorporated  into  the  next  units  on  the  production  line;  the  units 
already  produced  are  retrofitted  at  a  later  time.  Intuitively,  the  units  with  the  configuration 
change  should  initially  have  a  different  learning  rate  than  the  units  without  a  change 
because  the  manufacturers  must  learn  how  to  incorporate  the  change  into  the  production 
process.  However,  because  the  learning  rate  for  the  new  configuration  is  unknown,  DoD 
program  offices  generally  do  not  treat  the  reconfigured  units  with  a  different  learning  rate.  As 
a  result,  the  program  often  experiences  substantially  different  hours/costs  for  the  newly 
configured  production  units  than  their  original  learning  curve  projected.  A  configuration 
change  in  a  production  program  does  necessitate  learning  for  the  contractor,  and  the  impact 
to  learning  attributable  to  the  configuration  change  should  be  understood  by  all  levels  of  the 
DoD  acquisition  community.  Wright  (1936)  understood  this  limitation  to  the  learning  curve 
theory  application  even  in  the  infancy  of  the  idea: 

The  tremendous  cost  of  changes  introduced  into  a  production  order  during 
construction  is  too  well  known  to  require  emphasis.  This  cost  is  involved,  not 
only  in  shop  delays,  but  in  the  engineering  expense  of  re-designing.  It  is 
appreciated  that  in  a  rapidly  moving  art  such  as  aviation,  changes  are  more 
or  less  inherent.  ...  In  using  the  curve  developed  in  this  paper,  it  should  be 
recognized  that  the  factors  derived  are  based  on  the  assumption  that  no 
major  changes  will  be  introduced  during  construction.  (Wright,  1936,  p.  124) 

One  of  the  first  and  most  recognizable  learning  curve  formulas  is  y=  [ax]  Ab.  This  is 
referred  to  as  the  Unit  Learning  Curve  Model, 


y=  [ax]]  Ab 


(1) 


where 

y  =  the  estimated  production  hours  or  cost 
a  =  the  production  hours  of  the  theoretical  first  unit 
x  =  is  the  unit  produced 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-471  - 


b  =  is  a  factor  of  the  learning  =  log  R  (learning  rate)/log2 

The  Air  Force  methodology  on  learning  curves  and  guidance  in  their  application  is 
found  in  Chapter  8  of  the  Air  Force  Cost  Analysis  Handbook  (AFCAH)  and  Chapter  17  of  the 
DoD  Basic  Cost  Estimating  Guidebook  (BCE).  These  two  guides  focus  on  two  theories:  unit 
theory  and  cumulative  average  theory.  The  unit  theory,  Equation  1 ,  predicts  a  specific  unit 
cost.  Cumulative  average  theory  focuses  on  the  average  of  all  units  produced  up  to  a  certain 
point  in  the  production  process.  The  cumulative  average  and  unit  theory  have  been  the 
standard  in  manufacturing.  However,  research  has  shown  other  models  may  provide  a  more 
accurate  predictor  of  cost. 

Study  1 :  Production  Break  and  Lost  Learning 

Current  DoD  program  office  cost  estimating  assumes  a  stable  rate  of  learning  once  a 
program  is  substantially  into  production.  However,  intuitively,  a  configuration  change 
introduced  into  the  production  line  will  initially  disrupt  the  learning  effect.  This  study  will 
research  two  main  questions  to  address  the  implications  when  a  configuration  change 
occurs  during  production: 

1 .  Is  there  an  impact  to  the  learning  curve  slope  when  a  configuration  change  is 
introduced  to  the  production  line?  Specifically, 

a.  What  is  the  learning  curve  slope  for  each  new  configuration; 

b.  Are  the  production  segments  for  each  configuration  significantly 
different;  and 

c.  What  is  the  difference  between  the  hours  predicted  based  on  the  prior 
configuration  and  actual  hours  for  each  segment? 

2.  How  many  units  of  the  newly  configured  aircraft  are  produced  before  the 
contractor  recovers  the  stable  learning  rate? 

The  first  research  question  leads  to  a  single  testable  hypothesis  to  determine  if  the 
mean  amount  of  labor  hours  prior  to  a  configuration  change  is  the  same  as  the  mean 
amount  of  labor  hours  subsequent  to  a  production  change? 

Hypothesis  1 : 

Ho:  Mean  labor  hours  prior  to  configuration  change  —  Mean  labor  hours  post  configuration  change 

Ha:  Mean  labor  hours  prior  to  configuration  change  +  Mean  labor  hours  post  configuration  change 

If  the  analysis  results  fail  to  reject  the  null  hypothesis,  this  would  indicate  that  the 
data  points  come  from  the  same  population  and  a  configuration  change  did  not  have  a 
significant  impact  to  the  learning  during  production.  If  the  analysis  rejects  the  null 
hypothesis,  this  would  indicate  the  opposite,  that  the  data  points  representing  different 
configurations  come  from  different  populations  and  that  a  configuration  change  did  have  a 
significant  impact  to  the  learning  during  production.  If  the  results  support  rejecting  the  null 
hypothesis,  using  the  prior  learning  curve  equation  is  inappropriate  to  predict  the  hours  of 
the  new  configuration  because  the  units  come  from  different  populations.  The  second 
research  question  does  not  require  a  hypothesis  test. 

Production  Break  and  Lost  Learning 

As  the  learning  curve  theory  has  evolved,  researchers  and  practitioners  have 
investigated  the  impact  to  the  learning  rate  when  other  than  constant  production  exists. 
George  Anderlohr  (1969)  is  credited  with  developing  a  model  to  determine  the  additional 
hours/costs  that  result  from  a  break  in  production.  Anderlohr  (1969)  defined  a  production 
break  as  “the  time  lapse  between  completion  of  a  contract  for  the  manufacture  of  certain 
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units  of  equipment  and  the  commencement  of  a  follow-on  order  for  identical  units”  (p.  16).  A 
break  in  production  results  in  increased  hours  and  costs  because  the  laborers  are  no  longer 
performing  their  tasks  on  a  constant  repetitive  basis  and  the  laborers  become  less  efficient 
(have  a  loss  of  learning)  during  the  production  break  timeframe  (Anderlohr,  1969,  pp.  16- 
17). 

Studies  have  also  discovered  that  lost  learning  can  be  a  result  of  forgetting  at  times 
other  than  during  a  production  break  (which  is  considered  scheduled  forgetting).  Two  other 
instances  when  forgetting  can  occur  are  (1)  at  random  due  to  the  inability  to  continue  work 
(e.g.,  machine  breakdowns),  and  (2)  based  on  a  natural  process  (e.g.,  aging  workforce; 
Badiru,  1995,  p.  780).  Badiru  goes  on  to  conclude  that  “whenever  interruption  occurs  in  the 
learning  process,  it  results  in  some  forgetting.”  The  amount  of  forgetting  is  a  function  of  both 
the  length  of  disruption  and  the  initial  performance  level  (Badiru,  1995,  p.  780). 

Additional  Work  Theory 

A  similar  circumstance  to  the  production  break  theory  that  has  a  similar  result  is  the 
idea  of  new  learning,  when  manufacturing  is  interrupted  with  a  major  configuration  change 
to  the  production  unit.  When  the  unit  being  manufactured  is  changed,  the  laborers  must 
adjust  their  processes  to  learn  how  to  correctly  produce  the  newly  configured  unit. 
Historically,  adjusting  the  learning  curve  to  account  for  the  impact  due  to  configuration 
changes  is  referred  to  as  splicing  or  splitting  the  curve,  although  little  research  has  been 
done  in  this  area  with  empirical  data.  The  theory  of  splitting  the  curve  provides  rationale  to 
split  the  curve  between  units  of  different  configurations  (pre-  and  post-configuration  change) 
because  the  latest  production  unit  usually  provides  the  greatest  estimate  for  future 
production  units  (Dahlhaus  &  Roj,  1967,  p.  16). 

Sample  Data 

There  were  three  limiting  conditions  the  data  had  to  satisfy  to  be  included  in  this 
study:  (1)  at  least  one  identified  configuration  change  must  come  into  the  production  line 
during  production,  (2)  all  units  must  be  produced  on  the  same  production  line,  and  (3)  the 
program  must  be  “substantially”  into  production.  For  the  purposes  of  this  analysis, 
substantially  into  production  is  defined  as  those  units  considered  by  the  program  office  to  be 
representative  of  stable  production  and  exclude  any  units  identified  as  developmental  or 
pre-production. 

After  excluding  programs  that  did  not  meet  the  research  conditions,  there  were  four 
data  sets  available  at  the  time  of  the  analysis,  including  one  joint  service  and  three  Air  Force 
aircraft  programs.  Due  to  the  proprietary  nature  of  the  production  data,  the  program  names 
are  not  disclosed  and  will  be  identified  as  Programs  A,  B,  C,  and  D.  In  addition,  three 
classes  of  aircraft  are  represented  in  this  study:  Unmanned  Air  Vehicle,  Cargo,  and  Fighter 
aircraft. 

Analysis  Methods  and  Results 

To  test  the  research  hypothesis,  the  data  will  be  split  into  separate  segments  at  each 
identified  configuration  change  to  identify  if  the  segments  are  statistically  similar  based  on 
the  mean  or  median  labor  hour  values.  Using  the  touch  labor  hours,  the  learning  rate  before 
an  identified  configuration  change  and  the  learning  rate  after  the  change  will  be  calculated  to 
address  the  remaining  areas  of  the  first  research  question.  Both  calculations  will  use 
Crawford’s  unit  theory  equation  y  =  axb\  because  the  data  is  available  in  units,  a  unit 
analysis  is  appropriate.  In  addition,  to  avoid  the  smoothing  effect  and  the  obfuscation  of  unit 
variation  a  cumulative  unit  curve  can  create,  the  unit  learning  curve  method  will  provide  the 
most  explanatory  results  of  the  two  methods  (unit  and  cumulative  average)  for  the  intent  of 
this  study  (ICEAA  Module  7,  2013,  p.  14). 
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The  slope  will  be  calculated  each  time  an  identified  configuration  change  occurs  and 
not  at  other  instances,  even  if  a  pattern  change  is  evident  in  the  scatter-plot  of  the  data.  The 
learning  curve  equation  of  a  segment  will  forecast  the  touch  labor  requirements  of  the 
successive  production  segment.  The  forecasted  hours  of  an  identified  configuration  change 
will  be  compared  to  the  actual  hours  of  the  same  configuration  to  calculate  the  difference 
and  the  percentage  difference. 

To  answer  the  second  research  question,  an  analysis  will  determine  the  number  of 
aircraft  produced  after  a  configuration  change  until  the  prime  contractor  was  able  to  return  to 
a  stable  learning  rate.  This  will  be  accomplished  by  removing  one  production  unit  at  a  time 
(in  sequential  order  beginning  with  the  first  unit  of  the  segment)  and  calculating  the  learning 
curve  slope  of  the  remaining  units  until  the  stable  rate  of  the  prior  segment  is  achieved.  An 
overall  commonality  is  not  expected  because  every  program,  every  contractor,  and  the 
associated  production  process  are  different.  Instead,  the  results  are  informational  and  may 
support  contract  negotiation  efforts  with  more  insight  into  post-configuration  change 
production. 

Table  1  includes  the  slope  calculations  for  each  program  for  each  segment  identified. 
Configuration  A  is  always  the  initial  configuration,  prior  to  any  changes.  Based  on  this 
summary,  the  slope  never  remained  the  same  after  a  configuration  change.  Program  C  and 
Configuration  C  of  Program  B  were  excluded  from  the  analysis  at  this  point  based  on  the 
inappropriateness  of  the  analysis  method  displayed  through  scatter-plots  in  both  unit  and  log 
space. 


Table  1.  Segment  Learning  Curve  Slope  Values 


Configuration 

A 

B 

c 

D 

E 

F 

E 

ffl 

A 

63.26% 

49.84% 

- 

- 

- 

- 

1 _ 
to 

Q 

B 

87.02% 

84.33% 

- 

- 

- 

- 

a 

D 

91*96% 

73.15% 

59.04% 

68.56% 

75.24% 

82.49% 

In  nearly  every  case  involved  in  this  study,  the  segmented  data  are  statistically 
different  when  compared  to  an  adjacent  segment,  which  addresses  the  issue  in  the  first 
research  question  (Configurations  A  and  B  of  Program  B  were  not  found  to  be  statistically 
different).  There  is  a  change  to  the  learning  curve  slope  each  time  a  configuration  change  is 
introduced,  and  in  every  case  examined  except  one,  the  median  labor  hours  (which  are 
partially  a  function  of  the  learning  curve  slope)  for  the  different  configurations  are  statistically 
different.  These  findings  suggest  that  using  the  prior  learning  curve  equation  is  inappropriate 
to  predict  the  hours  of  the  new  configuration  because  the  units  come  from  different 
populations. 

Further  addressing  the  first  set  of  questions,  the  learning  curve  regression  equation 
for  each  segment  is  used  to  predict  the  touch  labor  hours  for  each  unit  in  the  following 
segment.  The  total  predicted  hours  for  each  segment  are  compared  to  the  total  actual  hours 
of  the  segment,  and  the  results  are  shown  as  a  difference  in  hours  as  well  as  a  percent 
difference  for  comparison  between  the  programs.  A  negative  value  indicates  the  estimate 
was  lower  than  the  actuals.  Table  2  details  the  results  of  the  predicted  and  actual  hour 
comparisons. 
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Table  2.  Learning  Curve  Equation  Prediction  vs.  Actuals  Summary 


Program  A 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B 

11,336,756.40 

11,371,252.00 

(34,495.60) 

-0.30% 

Program  B 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B* 

229,114.62 

295,348.35 

(66,233.73) 

-22.43% 

Program  D 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B 

1,014,525.48 

986,331.30 

28,194.18 

2.86% 

B  predicting  C 

490,909.41 

531,988.54 

(41,079.13) 

-7.72% 

C  predicting  D 

339,726.00 

368,921.32 

(29,195.31) 

-7.91% 

D  predicting  E 

678,070.58 

698,789.63 

(20,719.06) 

-2.96% 

D  predicting  F 

397,530.17 

542,429.97 

(144,899.80) 

-26.71% 

•Configuration  B  not  considered  a  statistically  significant  change  from 
configuration  A 

Given  that  this  portion  of  the  analysis  only  includes  three  programs,  and  two  of  the 
programs  only  compare  two  segments,  there  are  too  few  data  points  to  develop  any 
meaningful  CER  or  factor.  However,  the  results  are  still  impactful  because  for  each  of  the 
seven  segment  comparisons,  no  fewer  than  20,000  hours  were  the  difference  between  the 
predicted  and  actuals,  which  equates  to  millions  of  dollars  per  segment  misestimated 
(generally  underestimated)  in  a  cost  estimate.  Underestimation  requires  the  program  office 
to  find  dollars  not  currently  in  its  budget,  and  overestimation  temporarily  ties  up  funding  that 
can  be  used  for  other  purposes. 

In  reality,  a  contractor  will  submit  a  tech-refresh  proposal  to  the  program  office  to 
account  for  the  configuration  change,  but  will  estimate  the  unit  costs  based  on  an 
extrapolation  of  its  stable  learning  curve  because  the  new  slope  is  unknown.  In  every 
program  analyzed  in  this  study,  the  learning  curve  slope  becomes  much  steeper  after  the 
configuration  change  (when  compared  to  the  initial  stable  slope),  and  a  extrapolation  of  the 
stable  curve  will  create  a  higher  per  unit  cost  than  the  contractor  would  actually  experience 
with  the  steeper  learning  curve.  This  phenomenon  is  explored  in  the  next  section,  which 
analyzes  Program  A  to  answer  the  second  research  question  of  how  many  production  units 
are  manufactured  before  the  contractor  returns  to  its  stable  learning  rate. 

Program  A  was  selected  for  analysis  in  addressing  the  second  research  question 
because  Program  A  has  a  large  sample  size  in  total  and  within  each  segment.  In  addition, 
only  one  configuration  change  came  into  the  production  line,  so  this  program  provides  a 
consistent  sample  to  analyze.  The  stable  slope  for  Program  A  is  63.26%  as  determined  by 
the  units  in  Configuration  A  (units  41  to  71 ).  Table  3  summarizes  the  slopes  for 
Configuration  B  beginning  with  units  72  to  124  and  removing  one  unit  at  a  time  from  the 
beginning  of  the  segment  until  the  stable  slope  was  reestablished. 
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Table  3.  Program  A  Stable  Slope  Analysis  Summary 


unit 

Slope 

Units  to  Stabilize 

72 

49  £4% 

73 

50.59% 

X 

74 

51.34% 

2 

75 

51  J5% 

1 

76 

52  45% 

4 

77 

52  35% 

5 

78 

52  .83% 

6 

79 

52  3  1% 

7 

80 

51.21% 

3 

82 

53  44% 

9 

82 

53  -80% 

10 

S3 

54,16% 

11 

84 

54.35% 

12 

85 

55 .25% 

13 

88 

55.31% 

14 

87 

57  -39% 

15 

88 

59-15% 

16 

S3 

60.52% 

17 

90 

82  01% 

13 

91 

51 .50% 

19 

92 

54 .25  % 

93 

54  30% 

94 

€4.52% 

95 

63,51% 

95 

62.06% 

97 

50  34% 

The  stable  learning  rate  is  achieved  with  the  production  of  unit  91,  which  is  19  units 
after  the  configuration  change  came  into  the  production  line.  While  every  program  will 
stabilize  at  different  production  rates,  the  important  point  in  this  analysis  is  that  after  the 
configuration  change  is  introduced,  the  contractor  learns  much  quicker  on  the  units  after  the 
configuration  change  than  the  stable  flatter  learning  rate  pre-change.  The  units  immediately 
following  the  stabilized  rate  (92  to  97)  are  included  in  the  table  to  show  that  the  contractor 
does  not  continue  to  learn  for  all  units  after  the  stabilized  rate  is  achieved;  rather,  the 
contractor’s  learning  rate  stays  around  the  stabilized  rate.  While  this  analysis  is  for  only  one 
program  and  cannot  be  generalized  for  all  programs,  the  prior  analysis  did  show  that  for 
each  program,  the  contractor  learned  at  a  much  steeper  rate  following  the  configuration 
change.  These  results  provide  evidence  to  support  a  position  other  than  the  contractor 
extrapolating  the  prior  stable  learning  curve  in  a  tech-refresh  proposal  before  a  configuration 
change  is  introduced. 

Conclusions  of  Research 

The  hypothesis  testing  indicated  a  statistically  significant  difference  in  the  median 
production  touch  labor  hours  in  the  pre-configuration  change  and  post-configuration  change 
aircraft  for  every  pair  of  data  segments  analyzed,  except  for  one.  Comparing  the  median 
values  may  equate  to  a  statistically  significant  difference  in  the  learning  curve  slopes  for 
those  data  segments  because  the  impact  to  the  learning  curve  slope  is  evaluated  through 
the  touch  labor  hours  of  the  data  points,  as  they  are  partially  a  function  of  the  slope  value. 

The  data  point  analysis  to  address  the  stable  learning  curve  research  question 
produced  interesting  results.  The  analysis  did  show  a  pattern  that  post-configuration 
change,  the  contractor  initially  learns  at  a  much  faster  rate  and  the  learning  rate  decreases 
with  each  subsequent  unit  until  the  stable  learning  rate  is  again  achieved.  The  learning  rate 
did  appear  to  stabilize  at  this  point  and  did  not  continue  to  decrease. 

While  sample  size  was  limited  to  a  few  programs,  the  results  of  this  study  may  imply 
two  things.  First,  that  a  majority  of  the  time  there  is  an  impact  to  the  learning  curve  slope 
whenever  a  configuration  change  is  introduced  during  production.  Second,  that  the 
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contractor  is  able  to  learn  to  incorporate  the  change  much  more  quickly  than  its  stable 
learning  rate  for  the  entire  aircraft. 

Significance  of  Research 

The  results  of  this  research  indicate  there  may  be  a  significant  impact  to  the  learning 
curve  slope  when  a  configuration  change  is  introduced  during  production,  even  if  the 
program  is  substantially  into  production,  as  were  the  programs  included  in  this  analysis.  The 
findings  suggest  more  research  in  this  area  is  important  for  two  reasons.  First,  if  more 
programs  are  examined,  additional  data  points  may  lead  to  the  development  of  a  CER  or 
factor  to  adjust  a  stable  learning  curve,  which  would  be  a  useful  tool  for  cost  estimators 
given  the  ever-changing  acquisition  environment.  Second,  because  the  learning  curve  slope 
is  such  a  crucial  factor  in  production  contract  negotiations,  empirical  evidence  strengthens 
the  DoD’s  position  of  what  the  contractor’s  expected  learning  curve  should  be — which  this 
study  found  is  not  the  same  as  the  extrapolation  of  the  contractor’s  stable  learning  curve. 

An  initial  estimate  that  does  not  anticipate  any  configuration  changes  will 
underestimate  unit  production  hours  or  costs  required  for  the  newly  configured  unit.  If  the 
DoD  negotiates  a  contract  based  on  an  extrapolation  of  the  contractor’s  stable  rate,  these 
results  provide  evidence  that  the  stable  rate  will  overestimate  the  production  requirements; 
this  analysis  showed  the  contractor  learns  at  a  steeper  rate  after  a  configuration  change. 

The  initial  underestimating,  coupled  with  the  contractor’s  overestimation,  will  result  in  the 
program  office  requesting  millions  of  dollars,  possibly  in  excess,  per  configuration  change. 

Learning  curve  theory  advises  the  use  of  the  most  recent  or  most  representative 
production  methods  to  predict  the  follow-on  articles.  While  this  is  intuitive  and  proven  to 
result  in  better  estimates,  program  offices  cannot  disregard  the  prior  units.  If  program  offices 
track  the  configuration  change  information  and  the  resulting  impacts,  the  DoD  may  be  in  a 
better  position  to  estimate  costs  and  negotiate  production  contracts. 

Study  2:  A  Comparative  Study  of  Learning  Curve  Models 

History  shows  that  there  is  a  flattening  effect  near  the  end  of  production  runs, 
learning  does  not  remain  constant  in  aircraft  production,  and  machinery  is  becoming  more 
involved  in  the  production  process.  There  is  evidence  to  support  a  hypothesis  that  a  different 
model  may  be  more  accurate  than  Wright’s  model.  Prior  research  establishes  the  foundation 
for  further  research  into  additional  types  of  aircraft.  There  is  evidence  to  support  a 
hypothesis  that  a  different  model  may  be  more  accurate  than  Wright’s  model.  Previous 
research  found  that  the  contemporary  models  are  more  accurate  than  Wright’s  model  given 
an  incompressibility  factor  (M)  that  is  somewhere  between  0.0  and  0.1.  M  is  a  number 
between  zero  and  one  where  zero  indicates  a  completely  manual  process  and  one  indicates 
a  fully  automated  process.  Wright’s  model  is  the  most  accurate  predictor  of  cost  if  M  is 
assumed  to  be  greater  than  0.1.  Specifically,  further  research  and  analysis  using  program 
integration,  assembly,  and  checkout.  Additional  research  on  the  impressibility  factor  may 
indicate  a  model  that  is  more  applicable  to  DoD  methodology.  For  the  purpose  of  this  study, 
the  contemporary  models  examined  are  the  DeJong  and  S-Curve  Models.  The  following 
investigative  questions  are  the  basis  for  study  2: 

1 .  How  does  the  application  of  learning  curve  models  using  program  integration, 
assembly,  and  checkout  data  affect  learning  curve  models  that  incorporate  an 
incompressibility  factor? 

2.  How  sensitive  are  IA&CO  data  to  the  incompressibility  range? 

3.  Which  learning  curve  model  is  the  most  accurate  at  predicting  cost  or  time? 
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4.  How  can  the  individual  airframe  work  codes  prove  beneficial  for  predicting 
cost  or  time? 

The  questions  lead  to  the  following  hypotheses: 

HI :  One  of  the  compared  models  will  have  Mean  Absolute  Percent  Error 
significantly  different  from  the  others. 

H2:  One  of  the  modern  learning  curve  models  will  be  significantly  more 
accurate  than  Wright’s  model  in  predicting  costs  or  hours. 

H3:  The  S-Curve  model  with  IA&CO  will  have  a  significantly  lower  MAPE  than 
Wright’s  and  DeJong’s  Learning  Curve  Models. 

H4:  The  incompressibility  factor  will  have  a  significant  influence  on  the 
accuracy  of  the  DeJong  and  S-Curve  Models. 

The  first  hypothesis’  null  ( H0 )  is  /h  =  n2  =  /i3.  This  means  that  the  MAPE  (lower 
MAPE  is  better)  for  each  learning  curve  model  is  the  same.  The  alternative  hypothesis(//a) 
is  that  one  of  the  model’s  MAPE  is  statistically  different.  A  rejection  of  the  null  hypothesis  in 
favor  of  the  alternative  hypothesis  supports  significant  finding.  The  significant  finding  means 
that  testing  each  contemporary  learning  curve  model  against  Wright’s  model  is  the  next 
phase.  The  second  hypothesis  (H2)  has  a  null  jux  =  fa,  where  i  will  equal  models  2  and  3 
(DeJong’s  and  the  S-Curve).  The  Ha  is  that  the  contemporary  learning  curve  models  will 
have  a  lower  MAPE  than  the  conventional  models  >  fa.  H3’s  null  hypothesis  is  fa  = 

The  Ha  is  that  fa  >  M 3,  meaning  that  the  S-Curve  will  have  the  lowest  MAPE  and  thus  be  the 
most  accurate  predictor  of  cost  or  hours.  The  last  hypothesis  is  that  small  changes  in  the 
incompressibility  factor  will  have  a  large  influence  on  the  MAPE  of  each  model. 

Relevant  Learning  Curve  Research 

Relevant  research  has  highlighted  an  important  point  in  why  military  programs  have 
not  adapted  a  contemporary  learning  curve  model: 

Because  of  the  regularity  of  production  in  military  programs,  organizational 
forgetting,  and  spillovers  of  production  experience  are  less  apparent.  If 
forgetting  is  present,  it  may  be  very  difficult  to  identify  (e.  g.,  data  could  be 
consistent  with  either  a  20  percent  learning  rate  or  a  25  percent  learning  rate 
with  5  percent  forgetting).  And,  in  most  cases  there  are  not  many  model 
variants,  so  spillovers  are  not  important.  (Benkard,  1999,  p.  4) 

The  newest  fighter  weapon  in  the  U.S.  military  arsenal  will  be  the  F-35.  The  F-35  has 
three  variants,  and  the  Pentagon  plans  to  spend  over  $390  billion  on  these  aircraft  (Luce, 
2014).  Five  percent  of  $390  billion  attributed  to  learning/forgetting  processes  is  still  a 
staggering  number.  The  point  is  that  there  is  room  for  improvement.  Many  of  the  fighter 
aircraft  in  use  today  have  had  multiple  models.  The  F-15  had  models  A-E,  and  the  F-16  had 
models  A-F.  The  DoD  can  use  the  hypothesized  5%  forgetting  to  save  millions  of  taxpayers’ 
dollars.  The  accurate  estimates  result  in  less  spending,  or  savings  that  could  go  into  other 
taxpayer  needs  or  public  works.  The  estimates  enable  the  DoD  to  truly  forecast  budget  and 
spending  levels. 

One  must  consider  the  question  of  what  does  the  future  actually  look  like  in  regards 
to  machinery  and  automation?  Are  people  still  going  to  be  relevant  for  production?  In  the 
Defense  acquisitions  realm,  the  basis  is  that  with  low  purchase  quantities  for  state  of  the  art 
machines  will  not  rely  on  technological  or  machine  dominated  production.  This  idea  really 
comes  down  to  the  machine  verses  machines  argument.  Asking  whether  a  robot  will  take 
the  jobs  of  humans  is  key.  Experts  say  yes  and  no.  In  the  past,  machines  were  used  to 
replace  manual  labor  that  was  intensive  and  repetitive.  According  to  a  study  by  the  Bank  of 
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America,  robots  are  likely  to  be  performing  45%  of  manufacturing  tasks  by  2025,  versus 
10%  today  (Madigan,  2011).  The  price  of  a  computer,  a  robot,  a  chip,  etc.  is  falling,  and  it  is 
speculated  that  it  will  fall  even  more  in  the  future.  However,  the  jobs  that  require  human 
interaction  are  least  likely  to  be  replaced  by  a  robot.  Maybe  DoD  acquisitions  are  in  the  clear 
and  100%  of  learning  is  still  realizable  for  learning  curve  methodology.  Experts  agree  that 
the  future  does  include  a  significant  presence  of  machinery  because  the  prices  of  robotics 
and  computers  are  decreasing  while  the  cost  of  human  employees  are  increasing  (Aeppel, 
2012). 

Contemporary  Learning  Theory 

A  contemporary  variation  of  learning  curve  models  is  DeJong’s  Learning  Formula. 
This  formula  is  a  modification  of  Wright’s  model,  and  it  takes  into  account  the  constant,  M.  M 
is  the  incompressibility  factor,  which  is  a  constant  between  zero  (fully  manual  operation)  and 
one  (fully  automated  or  machine  dominated  operation;  Badiru  et  al.,  2013).  Equation  2 
highlights  DeJong’s  Learning  Model. 

tn  =  h{M  +  (1  -  M)n~b  (2) 

where 

tn=  the  cumulative  average  time  after  producing  n  units 
=  time  required  to  produce  theoretical  first  unit 
n  =  cumulative  unit  number 
b  =  log  Rl\og  2  (learning  index) 

M  =  incompressibility  factor  (a  constant) 

A  machine  based  production  process  would  result  in  no  learning,  and  thus  an  M 
value  of  one.  It  the  basis  of  this  thesis  and  belief  that  aircraft  production,  complex  in  nature, 
has  an  M  value  close  zero  because  aircraft  production  is  a  highly  manual  process.  Thus,  M 
would  be  closer  to  zero  for  IA&CO.  M  does  not  have  a  specific  value.  This  research  will 
focus  on  the  best  M  value  for  the  particular  aircraft  production.  A  potential  weakness  of  the 
DeJong  model  is  that  it  does  not  take  into  account  previous  units  produced  as  much  as  the 
S-Curve  model  does. 

The  S-Curve  Model  takes  into  account  both  previous  units  produced  and  the 
incompressibility  factor.  Figure  1  shows  the  effects  of  learning  over  time  as  hypothesized 
from  the  S-Curve  Model.  The  linear  nature  of  Wright’s  original  learning  curve  model  has 
been  in  question  for  many  years  (Everest,  1988).  The  Rand  Corporation  first  sought  to 
explain  the  progression  of  the  learning  curve  used  to  estimate  costs  for  both  military  and 
civilian  airframes.  The  report  attempted  to  describe  the  relationship  between  units  (quantity) 
and  costs  and,  ultimately,  whether  the  relationship  was  linear  on  a  log  scale.  The  results  of 
the  Rand  Corporation  found  that  a  convex  curve  may  provide  more  accuracy  if  producing  a 
large  number  of  units  (Asher,  1956).  The  results  found  that  a  convex  model  provides  less 
error  if  there  is  a  need  for  large  extrapolation.  Essentially,  an  estimation  of  significantly  more 
units  in  production  instead  of  fewer  provides  less  error.  For  units  where  large  extrapolation 
was  required,  a  non-linear  model  was  more  appropriate.  The  S-Curve  model,  convex  in 
nature,  presents  a  shape  of  learning.  The  S-Curve,  when  plotted  using  a  log  scale 
relationship,  follows  an  S  function.  The  experience  over  time  (attempts  at  learning)  may 
exhibit  the  S-Curve  (Everest,  1988). 
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Figure  1.  S-Curve  Learning  Model 

Initially,  there  is  a  slow  beginning  as  a  worker  learns  the  production  process.  The 
newness  of  the  product  is  a  characteristic  of  the  slow  initial  beginning.  New  tools,  methods, 
shortages  of  parts,  reworks,  and  the  challenge  of  developing  a  cohesive  production  team 
are  all  potential  contributors  to  the  slow  beginning.  The  fact  that  the  initial  stage  in 
production  deals  changes  from  tooling  to  even  workers  contributes  to  the  gradual  start 
(Badiru,  1992).  From  there,  learning  and  familiarity  of  tools,  methods,  and  workers  occur. 

The  learning  enables  a  steep  acceleration  of  production.  Production  improvement  occurs 
with  attempts  on  the  process,  or  learning  by  doing.  An  example  from  the  literature  is  aircraft 
production.  Aircraft  production  that  includes  workers  and  tools  that  are  more  efficient  leads 
to  an  assembly  process  that  is  also  more  efficient.  The  efficiencies  found  result  in  less  time 
to  complete  an  aircraft  (Asher,  1956).  However,  the  improvement  and  efficiencies  eventually 
begin  to  fade.  The  plateau  at  the  trailing  edge  of  the  curve  is  the  slope  of  diminishing  returns 
where  the  curve  begins  to  flatten  out,  or  in  many  occurrences  at  the  end  of  production 
cycles,  there  is  a  “tailup”  (Everest,  1988).  After  time,  inefficiencies  can  occur:  forgetting, 
experienced  workers  focusing  on  new  projects,  failure  to  repair  worn  tooling  at  the  normal 
rate,  increase  of  machine  disassembly,  lack  of  key  materials  (safety  stock),  and  workers 
taking  more  time  to  prolong  their  employment  (Everest,  1988).  The  S-Curve  equation  is 
shown  in  Equation  3: 

tn=  1 1  +  M{n  +  B)~b  (3) 

where 

tn=  the  cumulative  average  time  after  producing  n  units 
tx  =  time  required  to  produce  theoretical  first  unit 
n  =  cumulative  unit  number 
b  =  log  Rl log  2  (learning  index) 

M  =  incompressibility  factor  (a  constant) 

B  =  equivalent  experience  units  (a  constant) 

From  this  equation  and  Figure  1 ,  the  forgetting  concept  is  evident.  The  S-Curve 
portrays  that  with  time,  some  inefficiencies  will  occur.  Use  of  the  S-Curve  and  DeJong 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-480- 


Models  may  provide  more  precision  to  learning  curve  estimation  and  enable  higher  accuracy 
within  DoD  cost  estimating  because  they  include  influences  that  were  previously 
unaccounted  for. 

Data 

The  Air  Force  Life  Cycle  Management  Center  Cost  Staff  (AFLCMC/FCZ)  at  Wright- 
Patterson  Air  Force  Base  (WPAFB)  provided  learning  curve  data  for  17  Major  Defense 
Acquisition  Programs  (MDAPs).  A  MDAP  is  classified  as  a  major  program  that  exceeds  a 
certain  dollar  threshold.  There  are  80  MDAPs  in  the  DoD  as  of  2014.  The  numbers  have 
decreased  slightly  over  the  years.  The  data  files  consist  of  average  Learning  Curve  Reports 
of  Annual  Unit  Cost  (AUC)  in  addition  to  the  MDAPs  estimate  methods  using  Wright’s 
conventional  learning  curve  model.  Only  one  program  provided  was  broken  into  the  specific 
work  codes  that  include  the  needed  data  (IA&CO).  When  comparing  models  based  on 
airframe’s  integration,  assembly,  and  checkout  data,  the  assumption  of  incompressibility 
close  to  zero  is  acceptable  due  to  the  highly  individualized  process  completed  by  humans. 

A  description  of  labor  categories  Integration,  Assembly,  and  Checkout  highlights  that 
not  all  definitions  are  synonymous  amongst  manufacturers.  The  manufacturers  largely 
consider  what  is  involved  in  each  category  as  proprietary  information.  For  the  purpose  of 
this  study,  Final  Test  Integration,  Electrical  and  Mechanical  Assembly,  Test/Integration, 
Composites  (all  locations),  and  Quality  Control  are  considered  IA&CO.  Final  Test  Integration 
includes  the  direct  labor  for  the  final  integration  and  test,  which  includes  final  assembly, 
system  burn-in,  payload  integration  and  interface,  autopilot  checks,  taxi  tests,  range  tests 
and  first  flight  support.  Electrical  Assembly  is  the  direct  labor  required  to  assemble  electronic 
components.  Mechanical  Assembly  is  the  direct  labor  required  to  build  servos  for  the 
aircraft,  to  build  landing  gear,  build  starter/alternators,  to  perform  rework,  and  high-time 
maintenance  on  those  components.  Test/Integration  is  the  direct  labor  for  new  build 
electronics,  field  repairs,  integrating  avionics,  and  testing  them  at  the  system  level. 
Composites  manufacturing  is  the  direct  labor  required  to  lay  up,  cure,  and  finish  components 
such  as  the  fuselage,  wings,  tails,  and  landing  gear.  Quality  Control  is  the  direct  labor 
required  to  provide  inspection  of  electronics  and  mechanical  components  and  assemblies, 
document  discrepancies,  and  resolve  problem  areas.  Of  note,  these  labor  categories  involve 
mainly  direct  labor  performed  by  humans  where  the  learning  process  is  observable.  All  of 
the  data  includes  Test  Support,  Machine  Shop,  Program  Support  and  Design.  These  work 
codes  are  not  repetitive  in  nature  like  IA&CO. 

Analysis  Methods  and  Results 

The  data  includes  actual  costs  as  well  as  predicted  costs  using  one  of  the  learning 
curve  models.  Once  calculation  of  the  predicted  costs  is  complete,  the  error  is  simply  the 
difference  between  the  actuals  costs  and  the  predicted  costs.  To  provide  a  comparison,  a 
difference  calculation  in  the  absolute  value  and  absolute  value  percent  error  are  the  means 
of  analysis.  The  next  step  is  to  perform  an  analysis  to  test  the  hypotheses.  ANOVA  or  the 
Kruskal-Wallis  test  will  provide  the  basis  for  comparing  the  percent  errors.  The  tests  will 
produce  an  F-statistic  (a  test  statistic)  that  falls  within  a  Chi-distribution  and  a  p-value.  This 
comparative  study  will  produce  results  based  on  a  95%  confidence  level  (an  oc  of  0.05).  If 
the  P-value  is  less  than  0.05,  rejection  of  the  null  hypothesis  in  favor  of  the  alternative 
hypothesis  will  occur.  Rejecting  the  null  hypothesis  for  this  study  will  represent  that  there  is  a 
95%  chance  that  the  tested  populations  are  different.  The  conditions  for  ANOVA  are  as 
follows:  the  samples  must  be  from  a  random  selection  of  the  population,  normally 
distributed,  and  population  variances  must  be  equal.  If  the  conditions  for  ANOVA  fail  to  meet 
the  needed  criteria,  the  Kruskal-Wallis  test  (non-parametric  equivalent  to  ANOVA)  will  be  the 
test  to  determine  if  multiple  samples  arise  from  the  same  distribution  and  have  the  same 
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parameters  (“Kruksal-Wallis  Excel,”  n.d.).  The  ANOVA  or  Kruskal-Wallis  f-test  provides 
insight  into  the  first  hypothesis.  The  Kruskal-Wallis  test  is  beneficial  since  the  one-way 
ANOVA  is  usually  robust  based  on  the  assumptions  for  ANOVA.  The  Kruskal-Wallis  test 
becomes  useful  in  particular  when  group  samples  strongly  deviate  from  normal  (sample  size 
is  small  and  unequal  and  data  are  not  symmetric)  and  variances  are  different  (potential 
outliers  exist).  The  assumptions  for  the  Kruskal-Wallis  test  are  that  no  assumptions  are 
made  about  the  underlying  distribution;  however,  assume  that  all  groups  have  a  distribution 
with  the  same  shape,  and  no  population  parameters  are  estimated  (no  confidence  intervals 
in  the  data;  Zaiontz,  2015).  If  the  F-statistic  is  significant,  then  rejection  of  the  null 
hypothesis  in  favor  of  the  alternative  that  at  least  one  of  the  sample  means  is  different  is  the 
outcome. 

The  t-Test  for  two  samples  test  will  evaluate  the  second  hypothesis  that  one  or  more 
of  the  models  is  a  better  fit  to  the  data  than  Wright’s  Model.  The  control  for  this  comparison 
is  Wright’s  Learning  Curve  Model.  Since  Wright’s  Learning  Curve  Model  (WLC)  is  the  control 
for  this  study,  a  comparison  to  the  other  model’s  MAPEs  is  the  method.  If  the  assumption  for 
equal  variance  is  not  met,  the  t-Test  for  two  samples  assuming  unequal  variances  will  be 
used.  The  next  analysis  that  corresponds  to  H3  will  be  testing  which  model  is  most  accurate 
given  significant  results  for  more  than  one  model  from  H2.  Once  again,  the  paired  difference 
t-test  is  the  next  step.  A  paired  difference  experiment  uses  a  probability  distribution  when 
comparing  two  sample  means  and  produces  a  t-statistic  that  falls  within  a  student-t 
distribution  that  can  either  reject  or  fail  to  reject  the  null  hypothesis  depending  on  the  desired 
confidence  level.  Lastly,  H4  will  require  reiterations  of  the  tests  in  order  to  determine  a  good 
estimate  for  incompressibility  factor  based  on  the  airframes.  This  method  will  include  IA&CO 
and  then  all  of  the  data  in  order  to  provide  a  comparison. 

Of  note,  the  reader  may  question  why  the  means  cannot  provide  the  basis  for  the 
analysis.  This  lies  in  the  variation  of  the  means.  If  the  coefficient  of  variation  (standard 
deviation  as  a  percentage  of  the  mean)  is  low,  the  mean  may  be  a  good  predictor  of  the 
better  model.  However,  as  a  rule  of  thumb,  if  the  coefficient  of  variation  (CV)  is  greater  than 
15%,  the  mean  indicates  a  looser  distribution.  Most  analysts  would  likely  prefer  a  tighter 
distribution  with  less  variability.  In  practice,  a  low  CV  (say,  5%)  would  indicate  that  the 
average  (mean)  of  the  cost  data  is  a  useful  description  of  the  data  set.  On  the  other  hand,  if 
the  CV  is  much  higher  (say,  greater  than  15%),  there  should  be  a  cost  driver  in  the  data  set 
that  causes  the  cost  to  vary.  The  CVs  for  the  analysis  will  provide  insight  into  the  dispersion 
of  the  data  points.  CVs  for  the  analysis  all  exceeded  the  threshold  and  resulted  in  the  mean 
not  being  a  good  predictor.  Table  4  highlights  the  MAPE  analysis  for  M  =  0.05.  Results  were 
inconclusive  as  to  which  model  was  more  accurate. 


Table  4.  MAPE  Analysis  M  =  0.05 


WLC 

DeJong 

S-Curve 

4.11% 

3.00% 

2.64% 

A  description  of  the  results  for  the  Air  Force  Program,  based  on  the  assumption  of 
low  incompressibility  values  of  0.05  and  0.1,  highlights  the  effects  of  learning.  The  results 
changed  between  these  values  and  indicated  that  the  S-Curve  model  may  be  a  more 
accurate  predictor  at  0.05,  but  WLC  is  more  accurate  at  0.10.  After  0.10,  WLC  becomes 
significantly  less  error  prone  for  both  IA&CO  and  the  analysis  with  all  of  the  work  codes. 
WLC  is  a  better  predictor  of  cost  when  the  incompressibility  is  0.10  and  higher.  Analysis  on 
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all  of  the  data  points,  not  just  IA&CO,  showed  no  difference  between  the  models  at  an  M  of 
0.05.  However  increasing  the  M  value  results  in  WLC  becoming  an  increasingly  more 
accurate  predictor  of  cost.  The  following  figures  highlight  the  effects  of  M  and  the  Absolute 
Percent  Errors. 


When  plotting  the  Absolute  Percent  Errors  for  all  of  the  data  points,  a  trend  similar  to 
the  analysis  using  IA&CO  was  evident.  WLC  starts  as  a  more  accurate  predictor  of  cost  and 
then  becomes  less  and  less  accurate,  whereas  the  DeJong  and  S-Curve  models  become 
increasingly  more  accurate  predictors.  Figure  3  shows  the  results  of  APE  for  all  of  the  data. 
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Conclusions  of  Research 

In  summary,  results  support  the  first  hypothesis  that  there  is  a  significant  difference 
between  the  models.  Results  are  inconclusive  as  to  whether  any  models  are  significantly 
more  accurate  than  Wright’s  model.  Between  an  incompressibility  of  0  to  0.1,  DeJong  and 
S-Curve  models  were  more  accurate  (less  error  prone).  Nevertheless,  at  an 
incompressibility  of  0.10  and  beyond,  Wright’s  model  is  most  accurate.  The  third  hypothesis 
was  also  inconclusive  as  to  which  model  is  most  accurate  at  an  incompressibility  of  0.05. 

Both  DeJong  and  S-Curve  models  were  more  accurate  than  WLC,  but  there  was  no 
difference  between  the  two.  Finally,  incompressibility  was  highly  influential  as  hypothesized. 
The  results  of  the  findings  lead  to  questioning  why,  for  the  program  chosen, 
incompressibility  would  become  increasingly  more  error  prone  when  more  automation  is 
present.  In  addition,  the  findings  put  into  question  how  the  DoD  can  draw  a  conclusion  about 
the  application  of  contemporary  learning  curve  models  in  acquisitions  and  specifically  cost 
estimation.  Absolute  Percent  Error  figures  highlight  that  WLC  is  accurate  initially  and 
eventually  becomes  increasingly  less  accurate.  The  opposite,  S-Curve  and  DeJong  Models 
are  not  as  accurate  initially,  but  become  increasingly  accurate  over  trials.  The  MAPE 
analysis  averages  all  of  the  errors.  If  the  data  set  included  more  units,  results  may  trend 
towards  results  in  favor  of  the  contemporary  models.  That  answer  is  based  on  the  visual 
trend  from  the  APE  figures.  Of  interest  when  the  incompressibility  factor  is  0.10,  the  models 
portray  that  90%  of  learning  is  obtainable.  Because  the  data  set  was  small,  changes  in 
incompressibility  may  not  be  as  evident  to  the  significance  of  the  comparative  study. 

The  findings  also  put  into  question  how  the  DoD  can  draw  a  conclusion  about  the 
application  of  contemporary  learning  curve  models  in  acquisitions  and  specifically  cost 
estimation.  If  the  production  cycle  is  long  and  many  trials  will  be  realized,  there  is  potential 
that  the  contemporary  models  may  capture  a  more  accurate  picture  of  learning.  Aircraft 
production  may  provide  starkly  different  results  from  a  missile  production  run  where  more 
units  are  produced  over  time.  The  results  support  that  there  is  potential  for  a  more  accurate 
model.  However,  it  may  not  be  in  the  realm  of  aircraft  production.  Aircraft  production  may 
include  some  automation.  It  is  not  implausible  that  aircraft  production  is  95%  manual  and 
supports  an  M  factor  of  0.05.  The  contemporary  models  may  support  a  more  automated 
process  such  as  a  production  line  much  like  the  automobile  industry.  Prior  studies  and 
subject  matter  expert  opinion  support  that  aircraft  production  is  manual.  However,  there  is  a 
belief  that  more  automation  will  be  present  in  the  future. 

Significance  of  Research 

Results  from  the  analysis  show  that  there  is  reason  to  believe  Wright’s  Learning 
Curve  may  not  be  the  best  method  for  estimating  costs.  By  extrapolating  from  actuals,  the 
method  for  Wright’s  model  may  not  incorporate  enough  of  the  variability  of  learning.  The 
results  provide  evidence  that  Wright’s  Model  is  accurate  initially,  but  with  attempts  at 
learning  (trials)  the  amount  of  error  increases.  The  comparative  analysis  on  learning  curve 
models  provides  a  standalone  analysis  of  program  actuals.  The  conclusions  from  this  study 
are  that  there  is  potential  for  a  more  accurate  cost-estimating  model  and  that  the 
conventional  learning  curve  models  become  increasingly  less  error  prone  over  trials.  The 
DeJong  and  S-Curve  models  show  promise  as  a  way  to  improve  DoD  cost  estimating. 

The  results  of  the  research  do  not  support  all  of  the  hypotheses.  Results  did  confirm 
that  the  incompressibility  factor  was  highly  influential  for  both  the  S-Curve  and  DeJong 
models.  The  results  of  the  comparison  changed  drastically  with  a  small  change  in  the 
incompressibility  factor.  The  DeJong  and  S-Curve  models  were  both  more  accurate  than 
WLC,  but  there  was  no  difference  between  the  two.  This  finding  makes  it  challenging  to 
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simplify  the  results  given  the  uncertainty  of  incompressibility.  The  influence  of  machinery  in 

longer  production  cycles  is  a  valid  assumption  for  the  future.  The  influence  of  automation  in 

this  comparative  study  was  evident  by  the  absolute  percent  error  graphs. 
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Background 


The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 


Research  Questions  ^SFIT 

VP''  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow.  1 

1 .  Is  there  an  impact  to  the  learning  curve  slope  when  a 
configuration  change  is  introduced  to  the  production  line? 
Specifically: 

a.  What  is  the  learning  curve  slope  for  each  new  configuration? 

b.  Are  the  production  segments  for  each  configuration  significantly 
different? 

c.  What  is  the  difference  between  predicted  and  actual  hours  for  each 
adjacent  segment? 

2.  How  many  units  of  the  newly  configured  aircraft  are  produced 
before  the  contractor  regains  the  stable  learning  rate? 
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Methodology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 

•  Segment  “stable”  production  units 

•  Visual  Analysis 

•  Based  on  identified  configuration  changes 


Air  Force  Institute  of  Technology 


•  Nonparametric  Tests 

•  Compare  actuals  between  segments  to  determine  statistical  similarity 

•  Regression  Analysis 

•  Calculate  learning  curve  slope  for  each  segment 

•  Compare  predicted  versus  actual  hours  for  adjacent  segments 

•  Determine  number  of  production  units  to  stabilize  rate 
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4SFIT 


Program  A 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B 

11,336,756.40 

11,371,252.00 

(34,495.60) 

-0.30% 

Program  B 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B* 

229,114.62 

295,348.35 

(66,233.73) 

-22.43% 

Program  D 

Predicted  Hours 

Actual  Hours 

Difference 

%  Difference 

A  predicting  B 

1,014,525.48 

986,331.30 

28,194.18 

2.86% 

B  predicting  C 

490,909.41 

531,988.54 

(41,079.13) 

-7.72% 

C  predicting  D 

339,726.00 

368,921.32 

(29,195.31) 

-7.91% 

D  predicting  E 

678,070.58 

698,789.63 

(20,719.06) 

-2.96% 

D  predicting  F 

397,530.17 

542,429.97 

(144,899.80) 

-26.71% 

Configuration  B  not  considered  a  statistically  significant  change  from 
configuration  A 
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Data  Analysis 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 


Regression  Analysis 

•  Compare  predicted  versus  actuals  for  each  segment  pair 

•  Negative  difference  indicates  under-estimation 


Findings:  Research  Question  1 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 


Air  Force  Institute  of  Technology 


1 .  Is  there  a  significant  impact  to  the  learning  curve  slope  when  a 
configuration  change  is  introduced  to  the  production  line? 
Specifically: 

a.  What  is  the  learning  curve  slope  for  each  new  configuration? 


Configuration 

A 

B 

C 

D 

E 

F 

E 

CE 

A 

63.26% 

49.84% 

- 

- 

- 

- 

o 

B 

87.02% 

84.33% 

- 

- 

- 

- 

CL 

D 

91.96% 

73.15% 

59.04% 

68.56% 

75.24% 

82.49% 

b.  Are  the  segments  of  each  configuration  significantly  different? 

•  Each  segment  statistically  different  aside  from  Program  B  between 
configurations  A  and  B 

c.  What  is  the  difference  between  predicted  and  actual  hours  for  each 
adjacent  segment? 

•  At  least  20  thousand  hours  (usually  under-estimated) 
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Findings:  Research  Question  2 
Program  A 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow.  ■ 


Air  Force  Institute  of  Technology 


2.  How  many  units  of  the  newly  configured 
aircraft  are  produced  before  the  contractor 
regains  the  stable  learning  rate? 

•  Large  sample  size 

•  In  total 

•  In  each  segment 

•  One  configuration  change 

•  Isolated  impact 

•  Stable  slope:  63.26% 

•  Configuration  A 

•  Stabilized  after  1 9  newly  configured  units 


First  Unit 

Slope 

Units  to  Stabilize 

72 

49.84% 

73 

50.69% 

1 

74 

51.34% 

2 

75 

51.95% 

3 

76 

52.48% 

4 

77 

52.85% 

5 

78 

52.83% 

6 

79 

52.81% 

7 

80 

53.21% 

8 

81 

53.44% 

9 

82 

53.80% 

10 

83 

54.36% 

11 

84 

54.85% 

12 

85 

55.25% 

13 

86 

56.33% 

14 

87 

57.39% 

15 

88 

59.18% 

16 

89 

60.52% 

17 

90 

62.03% 

18 

91 

63.60% 

19 

92 

64.36% 

93 

64.30% 

94 

64.52% 

95 

63.51% 

96 

62.06% 

97 

60.54% 
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Significance  of  Findings  ^SFIT 

-«P|^  '  ^  ^  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow.  1 

•  Configuration  changes  introduced  during  production  may  cause  a  statistically 
significant  impact  to  the  unit  production  learning  rate  and  production  hours 

•  After  most  of  the  configuration  changes  analyzed,  the  contractor  achieved  a 
steeper  rate  of  learning  than  the  stable  rate 

•  Analysis  of  Program  A  indicated  the  contractor’s  learning  decreased  with  each 
subsequent  production  unit  until  eventually  stabilizing 

•  In  reality,  a  contractor  will  submit  a  tech-refresh  proposal  to  program  office 
to  account  for  configuration  change 

•  Estimated  based  on  extrapolation  of  stable  learning  curve 

•  In  every  program  in  this  analysis  (and  in  most  segments),  a  newly  configured 
aircraft  initially  experienced  a  higher  rate  of  learning 

•  An  extrapolation  of  the  stable  curve  will  result  in  a  higher  per  unit  cost  than  the 
contractor  would  actually  experience 
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Air  Force  Institute  of  Technology  4XFIT 

^ ^  V  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow.  1 

The  Impact  of  Learning 
Curve  Model  Selection 
and  Criteria  for  Cost 
Estimation  Accuracy  in  the 
DoD  -  Part  2 
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Background/Research  Question^AF  1 T 

'  ^  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 

Learning  curves  are  commonly  used  in  production  estimates 

•  Production  accounts  for  the  majority  of  total  Acquisition  Costs 


Mr.  Thomas  Henry  (OSD  CAPE)  on  modernization 

•  “Manufacturing  and  depots  are  becoming  as  automated  as  possible.  Learning 
curves  could  get  much  different  in  the  future  due  to  machines” 


Heightened  scrutiny  of  cost  estimates 

•  Budget  Control  Act  of  201 1  seeks  to  reduce  federal  deficit 

Is  the  current  DoD  methodology  is  outdated?  Are  alternative  models 
are  more  accurate? 

•  Wright’s  original  learning  curve  theory  (CUMAV)  was  formulated  in  1936. 
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Learning  Theory  ^SFIT 

'  ^  ^  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 

T.  P.  Wright  (1936)  theorized  that  as  a  worker  performs  a  task 
multiple  times,  the  time  required  to  complete  that  task  will  decrease 
at  a  constant  rate. 

•  Constant  percentage  decrease  for  doubling  quantity 

•  Wright’s  Learning  Curve  (WLC)  Model: 

b 

y  =  ax° 

Learning  is  a  human  phenomenon  occurring  in  manual  labor,  so  we 
should  expect  the  most  learning  to  occur  when  the  production  process 
involves  a  great  deal  of  touch  labor  and  little  automation 

DeJong’s  Learning  Formula 

•  Incorporates  percentage  of  process  that  is  automated  into  learning  models 

S-Curve  Model 

•  Incorporates  prior  experience  units  (prototypes)  and  percentage  of  process  that  is 
automated  into  learning  curves. 
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Analytical  Tests  ^SFIT 

^  Air  Force  Institute  of  Technology 

The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 


•  Use  Historical  data  to  determine  CUMAV  vs.  Unit  Theory 
•  Use  Regression  statistics  to  determine  validity  of  regression  models 


•  Compare  all  sample  means  to  determine  if  any  models  are  different 
•  Use  skewness,  kurtosis,  and  standard  deviation  to  determine  normality 

•  Compare  sample  means  to  determine  which  models  are  different  from 
WLC  status  quo 

•  Compare  means  of  S-Curve  and  DeJong  if  they  are  more  accurate  than 
WLC  to  determine  the  most  accurate  model 
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V  Data  Table  Example  (WLC)  4^FIT 

Aid  rnarr  Ikiotiti  i*rr  n  «r  Trruuni  nov 


The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 


Unit  (x) 

Actual 

Predicted 

Error 

Absolute 

Value  Error 

Absolute 

Value  Percent 

Error 

20 

17336. SI 

17871.04 

-534.22 

534.22 

0.03 

21 

17221.82 

17721.17 

-499.35 

499.35 

0.03 

22 

17041.33 

17579.45 

-538.12 

538.12 

0.03 

23 

16916.11 

17445.09 

-528.99 

528.99 

0.03 

24 

16854.60 

17317.41 

-462.81 

462.81 

0.03 

25 

16797.48 

17195.83 

-398.35 

398.35 

0.02 

26 

16710.27 

17079.81 

-369.54 

369.54 

0.02 

27 

16606.65 

16968.92 

-362.27 

362.27 

0.02 

•  MAPE  is  the  average  of  the  Absolute  Percent  Error 

•  MAPES  at  M  of  0.05 

•  WLC  =  4.11% 

•  DeJong  =  3.00% 

•  S-Curve  =  2.64% 
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Results:  APE  Graphs 

1  The  AFIT  of  Today  is  the  Air  Force  of  Tomorrow. 
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Significance  of  Findings 

44  FIT 

Air  Force  Institute  of  Technology 

•  There  is  potential  for  a  more  accurate  model  in  predicting  the 
effects  of  learning  within  DoD  acquisitions 

•  S-Curve  and  DeJong  models 


•  Sensitivity  of  results  and  uncertainty  of  incompressibility 
factor  make  it  difficult  to  simplify  the  results 

•  Findings  provide  a  proxy  to  future  research  and  open  a 
dialogue  for  change  within  DoD  learning  methodology 

•  The  influence  of  machinery  potentially  displayed  with  long 
production  cycle 
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